Skip to content

feat: pluggable widget engine v2#68

Open
engalar wants to merge 10 commits intomendixlabs:mainfrom
engalar:feat/widget-engine-v2
Open

feat: pluggable widget engine v2#68
engalar wants to merge 10 commits intomendixlabs:mainfrom
engalar:feat/widget-engine-v2

Conversation

@engalar
Copy link
Copy Markdown
Contributor

@engalar engalar commented Apr 1, 2026

Stacked on #67. PLUGGABLEWIDGET syntax, auto datasource/child slots, 20 widget templates, widget CLI.

Copy link
Copy Markdown
Collaborator

@ako ako left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

51K additions — this is a very large PR. It stacks on #67 (which stacks on #66), so all issues from both prior reviews carry forward. The new commit adds ~39K lines, of which ~28K are widget template JSON files.

Scope

This should be at least 3 separate PRs:

  1. Widget template JSON files (20 new templates) — bulk addition, easy to review separately
  2. Pluggable widget engine v2 (widget_engine.go, augment.go, loader.go, visitor/executor changes)
  3. Widget CLI commands (cmd_widget.go)

Bundling makes this effectively unreviewable at 51K lines.

Code quality issues

1. Silent nil return in deepCloneMap (augment.go)

deepCloneMap serializes to JSON then deserializes back. On error, it returns nil without signaling the caller. This causes cascading nils — cloned properties silently become nil, and downstream code gets mysterious failures instead of a clear error.

2. Unsafe type assertions throughout

Multiple places assume BSON structure shapes without checking. For example, createAttributeObject silently returns nil AttributeRef when the attribute path has fewer than 2 dots, instead of returning an error for invalid paths.

3. Entity context mutation (widget_engine.go)

entityContext is saved/restored around Build() but mutated in-place during DataSource mapping. If an error occurs partway through, the context is left corrupted. Should use an immutable context stack or pass context as a parameter.

4. Placeholder ID generation (augment.go)

Uses atomic.AddInt64 for a counter but ResetPlaceholderCounter() exists for testing — this is a race condition if tests run in parallel. The deterministic format also means IDs may collide across concurrent augmentations.

5. Mode selection swallows ambiguity (widget_engine.go)

When multiple modes have no condition, the first fallback is silently chosen with no warning. This hides accidental duplicate mode definitions that could produce wrong widget output.

6. Debug logging still present from PR #66

The log.Printf("SERIALIZE CHECK: timelineCustom Widgets: ...") debug code in writer_widgets_custom.go is still here from the base commits.

Missing error handling

  • buildAttributeObjects: No validation that the resolved attribute path actually exists in the domain model before creating a reference
  • Operations in widget_engine.go (opAssociation, etc.): Check if source is empty but don't validate the resolved context (e.g., EntityName being set)
  • containsPlaceholderID gives generic error messages — doesn't indicate which field has the unreplaced placeholder

Design concerns

JSON round-trip for deep clone

Using JSON marshal/unmarshal for deep cloning BSON structures is fragile and has performance implications. Custom BSON types or binary data won't survive the round-trip. A proper BSON deep-clone would be more robust.

Tight coupling to BSON

PluggableWidgetEngine receives and returns bson.D directly with no abstraction. This makes the engine untestable without real BSON structures and means any serialization format changes require rewriting the entire module.

Testing

  • sdk/mpr/roundtrip_test.go is a good addition (+212 lines) — this addresses the gap flagged in PR #66 review
  • However: no tests for deepCloneMap error paths, no tests for partial operation failures, no tests for mode fallback ambiguity, no tests for placeholder ID uniqueness under concurrency

Design docs committed

docs/plans/2026-03-30-widget-docs-and-customwidget.md and docs/plans/2026-03-30-widget-docs-generation-design.md are committed. Are these intended to live permanently, or should they be removed post-implementation? They describe planned work rather than documenting what was built.

Template files

20 new widget templates added under sdk/widgets/templates/mendix-11.6/. These are extracted from Studio Pro which is the right approach per the repo's guidelines. However:

  • Were all templates verified to include both type AND object fields as required by sdk/widgets/templates/README.md?
  • timeline.json shows +0 -0 (empty file?) — needs verification

Summary

The core widget engine design is reasonable, but the PR needs:

  1. Splitting into reviewable units
  2. Error handling hardened (especially deepCloneMap, attribute validation, context mutation)
  3. Debug logging removed
  4. Tests for error paths
  5. All issues from PR #66 and #67 reviews resolved first

This was referenced Apr 1, 2026
@engalar engalar force-pushed the feat/widget-engine-v2 branch 2 times, most recently from 5e85cc0 to 5a2bc46 Compare April 3, 2026 03:31
engalar pushed a commit to engalar/mxcli that referenced this pull request Apr 3, 2026
Review fixes:
- Remove 7 debug log.Printf calls + timelineCustom debug block
- deepCloneMap returns error instead of silent nil
- Mode fallback returns error for ambiguous multiple defaults
- createAttributeObject validates path format upfront
- Association mapping validates EntityName is set

Engine migration:
- Add attributeObjects operation for Attributes: [...] → BSON Objects
- Create .def.json for TEXTFILTER/NUMBERFILTER/DROPDOWNFILTER/DATEFILTER
- Delete ~200 lines of hardcoded filter builders
- Add DROPDOWNSORT full-stack (lexer/parser/describe/def.json)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@engalar
Copy link
Copy Markdown
Contributor Author

engalar commented Apr 3, 2026

Review Response

Rebased on latest main and addressed all items. Changes in commit 5a2bc46.

Fixed

# Issue Fix
1 Debug logging (7 log.Printf + timelineCustom block) Removed all, including unused "log" import
2 deepCloneMap silent nil Returns (map[string]any, error), all callers propagate
3 createAttributeObject silent nil for invalid paths Validates Module.Entity.Attribute format upfront, returns error
4 Mode selection swallows ambiguity Returns error when multiple modes have no condition
5 opAssociation missing EntityName validation resolveMapping validates EntityName before Association operation

Engine migration (beyond review scope, but reduces hardcoded code)

Migrated 4 filter widgets + 1 sort widget from hardcoded builders to widget engine:

  • New attributeObjects operation handles Attributes: [Title, Code] → BSON Objects array
  • Created .def.json for TEXTFILTER, NUMBERFILTER, DROPDOWNFILTER, DATEFILTER, DROPDOWNSORT
  • Deleted ~200 lines of hardcoded filter builders (buildTextFilterV3, buildNumberFilterV3, buildDropdownFilterV3, buildDateFilterV3, applyFilterWidgetProperties, buildAttributeObjects)
  • Added DROPDOWNSORT full-stack: lexer token, parser rule, DESCRIBE detection

Verified: build passes, tests pass, DESCRIBE→check→exec→DESCRIBE roundtrip correct, mx check clean (CE0642 is missing required attribute in test script, not engine issue).

Items evaluated but not changed

Item Reason
Entity context defer pattern Standard Go pattern — defer restores on error correctly
Placeholder atomic.Uint32 Reviewer said atomic.AddInt64 — actual code uses correct type
containsPlaceholderID messages Already precise: specifies widget, distinguishes type/object
timeline.json empty 51KB, 1703 lines — +0 -0 was git diff display artifact
JSON round-trip clone Template data is JSON-compatible by design, no BSON-specific types
BSON coupling Entire sdk/ layer operates on BSON directly — abstraction is YAGNI
Design docs in docs/plans/ Correct location for plans

PR scope

The 28K of the 51K additions are widget template JSON files (extracted from Studio Pro). The remaining code changes are now more focused after the filter builder cleanup (-200 lines).

Copy link
Copy Markdown
Collaborator

@ako ako left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (updated — 3 new fix commits since last review)

Good progress addressing the issues from my earlier review. Here's what changed and what remains.

Issues addressed

Original issue Status
Debug logging (log.Printf) Fixed — removed from widget_engine.go
deepCloneMap silent nil return Fixed — now returns (map, error), callers handle errors
deepCloneTemplate silent nil return Fixed — now returns (*WidgetTemplate, error)
clonePropertyPair silent nil return Fixed — now returns (map, map, error)
createAttributeObject silent nil for bad paths Fixed — returns (bson.D, error) with validation
Filter widget code duplication Fixed — 4 nearly-identical buildTextFilterV3/buildNumberFilterV3/etc. replaced with engine-based opAttributeObjects operation (-340 lines)
Filter widget def.json files added ✅ 6 new definition files for filter widgets + dropdownsort
generateDefJSON now generates property mappings ✅ New — auto-generates mappings from MPK property types
Association ordering dependency Fixed — two-pass approach ensures datasource before association, with test

Remaining issues

1. ANTLR visitor files still generated (+2,635 lines)

mdlparser_base_visitor.go and mdlparser_visitor.go are still added. These don't exist on main and aren't consumed by any code. Was the ANTLR config intentionally changed to generate visitors, or is this accidental?

Also: MDLLexer.interp, MDLLexer.tokens, MDLParser.interp, MDLParser.tokens show as deleted (-4,123 lines). These are normally part of the generated parser output. Were they intentionally removed? This may break ANTLR tooling that relies on them.

2. opAttributeObjects swallows errors

attrObj, _ := ctx.pageBuilder.createAttributeObject(...)
if attrObj != nil {
    objects = append(objects, attrObj)
}

createAttributeObject now returns errors (good), but the caller discards them with _. If an attribute path is invalid, the attribute is silently skipped instead of reporting the error to the user.

3. deepCloneTemplate error ignored in tests

clone, _ := deepCloneTemplate(original)

Three test sites ignore the error from deepCloneTemplate. Tests should check errors — t.Fatal(err) if it fails.

4. augmentFromMPK silently falls back

clone, err := deepCloneTemplate(tmpl)
if err != nil {
    return tmpl  // silently returns unaugmented template
}

This is better than nil, but a clone failure means the template may be mutated later (since augmentation modifies in-place). Consider logging the error.

5. Design docs still committed

docs/plans/2026-03-30-widget-docs-and-customwidget.md and docs/plans/2026-03-30-widget-docs-generation-design.md are still present. Are these intended as permanent documentation?

6. Still too large for practical review

50K additions across 69 files. The widget template JSONs alone are ~28K lines. Splitting templates into a separate PR would make this much more reviewable.

New code quality — generateDefJSON (commits 2-3)

The auto-generation of property mappings from MPK definitions is a nice improvement. The two-pass approach for association ordering is clean and well-tested. One minor issue:

case "boolean", "integer", "decimal", "string", "enumeration":
    m := executor.PropertyMapping{
        PropertyKey: p.Key,
        Operation:   "primitive",
    }

This maps "enumeration" to "primitive" operation — is that correct? Enumerations in Mendix are typically not simple primitives. If the primitive operation just sets a string value, it may work, but the naming is misleading.

Summary

The fix commits address the most critical issues (error handling, debug logging, code duplication). The remaining concerns are the ANTLR visitor files, swallowed errors in opAttributeObjects, and PR size. The core engine changes look solid.

@ako ako mentioned this pull request Apr 3, 2026
3 tasks
engalar and others added 8 commits April 3, 2026 15:25
PLUGGABLEWIDGET syntax with explicit properties, auto datasource,
auto child slots, TextTemplate attribute binding, and keyword properties.

Core changes:
- PLUGGABLEWIDGET 'widget.id' name (key: value) syntax
- CUSTOMWIDGET/TABCONTAINER/TABPAGE grammar tokens
- Auto datasource ordering (step 4.1, before child slots)
- Auto child slot matching by container name
- Object list auto-populate with Required filter
- TextTemplate {AttrName} parameter binding
- Widget lifecycle CLI (widget init/docs)
- DESCRIBE PLUGGABLEWIDGET output for generic widgets
- 20 new widget templates (mendix-11.6)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Review fixes:
- Remove 7 debug log.Printf calls + timelineCustom debug block
- deepCloneMap returns error instead of silent nil
- Mode fallback returns error for ambiguous multiple defaults
- createAttributeObject validates path format upfront
- Association mapping validates EntityName is set

Engine migration:
- Add attributeObjects operation for Attributes: [...] → BSON Objects
- Create .def.json for TEXTFILTER/NUMBERFILTER/DROPDOWNFILTER/DATEFILTER
- Delete ~200 lines of hardcoded filter builders
- Add DROPDOWNSORT full-stack (lexer/parser/describe/def.json)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
generateDefJSON now derives propertyMappings and childSlots from MPK
property definitions instead of returning a skeleton. This makes
`mxcli widget init` produce usable .def.json files out of the box.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Association operations require entityContext set by a prior DataSource
mapping. If MPK lists association properties before datasource,
validateMappings would reject the generated definition. Two-pass
generation now collects association mappings and appends them last.

Added test: TestGenerateDefJSON_AssociationAfterDataSource

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parser/serializer fixes:
- Serialize NoCase in SequenceFlow CaseValues (was silently dropped)
- Parse Size field for all microflow objects (was always 0;0)
- Parse and preserve BezierCurve control vectors in SequenceFlow
- Skip MicroflowParameter in ObjectCollection parser (prevent duplication)
- Sort translation languages for deterministic Text serialization

Test strategy change:
- Double roundtrip (parse→serialize→parse→serialize) verifies idempotency
- Original Studio Pro .mxunit baselines preserved as ground truth
- Tests catch serialization regressions without requiring full field parity

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- docker: skip chmod permission tests on Windows (os.Chmod unsupported)
- docker/diaglog: use USERPROFILE on Windows, HOME on Unix for home dir
  override in tests (os.UserHomeDir reads USERPROFILE on Windows)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Map iteration order is non-deterministic in Go. Sort language keys
before serializing Translations in writer_microflow.go,
writer_enumeration.go, and writer_pages.go to ensure idempotent output.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use mxbuild 11.6.4 in CI (matches widget templates, was 11.8.0)
- Restore COLON? in MODIFY ATTRIBUTE grammar rule (lost during rebase)
- Fix duplicate widget name when direct children exist with childSlots
  defined — skip auto-slot Phase 2 when applyChildSlots handles defaults

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@engalar engalar force-pushed the feat/widget-engine-v2 branch from bede5ca to 324ec4d Compare April 3, 2026 07:31
Widget templates embedded in mxcli may have fewer properties than the
mpk version bundled with mxbuild. Running update-widgets before check
ensures the widget definitions are synchronized, preventing CE0463.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants